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We study collective behavior of magnetic adatoms randomly distributed on the surface of a topo- 
logical insulator. As a consequence of the spin-momentum locking on the surface, the RKKY-type 
interactions of two adatom spins depend on the direction of the vector connecting them, thus inter- 
actions of an ensemble of adatoms are frustrated. We show that at low temperatures the frustrated 
RKKY interactions give rise to two phases: an ordered ferromagnetic phase with spins pointing 
perpendicular to the surface, and a disordered spin-glass-like phase. The two phases are separated 
by a quantum phase transition driven by the magnetic exchange anisotropy. Ferromagnetic ordering 
occurs via a finite-temperature phase transition. The ordered phase breaks time-reversal symmetry 
spontaneously, driving the surface states into a gapped state, which exhibits an anomalous quantum 
Hall effect and provides a realization of the parity anomaly. We find that the magnetic ordering 
is suppressed by potential scattering. Our work indicates that controlled deposition of magnetic 
impurities provides a way to modify the electronic properties of topological insulators. 



Topological insulators in three dimensions are a class of 
time-reversal-invariant materials characterized by gapless 
surface states with Dirac-like dispersion (for a review, see 
Refs. [TJ [5] and references therein) . These topologically 
protected states originate from the bulk band inversion 
induced by strong spin-orbit interactions. The effective 
low-energy Hamiltonian has the form of the Rashba spin- 
orbit coupling, 
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where a is the electron spin, v is the Fermi velocity, and 
n is the normal vector to the surface, chosen to be along 
z direction. The Dirac nature of the surface states is 
expected to manifest itself in interesting physical effects, 
including magnetoelectric effect [IHlj , large Kerr and uni- 
versal Faraday effects [H [5] . In addition, locking of spin 
and momentum on the surface [5] , evident from Eq. ([T]) , 
gives rise to electric charging of magnetic textures [7], 
and opens up new opportunities for spintronics applica- 
tions HHID1- 

The aforementioned physical effects and device appli- 
cations rely on the ability to induce perturbations that 
break time reversal symmetry and open up a gap in the 
surface states spectrum. A relevant perturbation has the 
form of a mass term for Dirac electrons, 



H\ = ma. 



(2) 



In principle, such a perturbation can be induced by de- 
positing magnetic films. However, such a method has a 
significant disadvantage of being irreversible, and likely 
inducing too strong magnetic fields that can completely 
destroy surface states. Thus, alternative methods are 
needed which would allow for a controllable and re- 
versible manipulation of the topological surfaces states. 

Here we theoretically explore the possibility of modi- 
fying electronic properties of the surface states by con- 
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FIG. 1: Phase diagram of magnetic adatoms. Inset: Magne- 
tization of spins on the topological surface, which are inter- 
acting via RKKY interactions, as a function of the exchange 
anisotropy 5 — J\\/J z . We find the position of the quantum 
critical point, S c w 1.3, from the condition that the magneti- 
zation is decreased by 50%. This is supported by the fluctua- 
tions of the magnetization, which exhibit a maximum at the 
conjectured transition point, S c ~ 1.3. Cluster of 9 spins was 
considered, and averaging was performed over 150 disorder 
realizations. 



trolled adsorption of magnetic adatoms. We study the 
collective behavior of adatom spins, determined by the 
RKKY-type interactions mediated by the topological sur- 
face states. We argue that depending on the magnetic 
anisotropy and exchange anisotropy of a single impurity 
spin, the ground state of many spins is either a spin glass, 
or a ferromagnetically ordered state with spins pointing 
perpendicular to the surface. The two phases are sepa- 
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rated by a quantum phase transition. The phase diagram 
of magnetic adatoms suggested in this paper is summa- 
rized in Fig. [T] 

In the ferromagnetic phase the average exchange field 
of impurities induces a spin polarization dependent mass 
of Dirac electrons, Eq. (J2j|, opening up a band gap in the 
spectrum of the surface states. The mass depends on the 
concentration of adatoms and their type, and therefore 
is tunable. We find that this state is favored in a large 
region of the phase diagram (see below), and persists 
up to several tens of Kelvin. In contrast, in the spin- 
glass phase the average value of spin is zero, therefore, 
on average the time-reversal symmetry is not broken, and 
thus the gap is absent. There is, however, an insignificant 
disorder broadening which induces finite density of states 
at the Dirac point [15] . Our work shows that deposition 
of magnetic impurities provides a route to controllably 
change the spectral and transport properties of surface 
electrons. 

We start with the analysis of the RKKY interactions, 
employing the T-matrix description. The interaction en- 
ergy of two impurity spins located at Ri t 2, respectively, 
is given by [12] 



fi ia = -T^Trln [l - i 2 (s)G (e, R)h(s)G (e, -R) 



In this expression R = i?2 — Ri, £ are fermionic Matsub- 
ara frequencies, 1 1.2(e) are the low-energy T-matrices of 
the impurities, and Go(e,0) is the unperturbed Matsub- 
ara Green's function of the surface electrons. The trace 
here is taken over the spin space. Note that there are 
also single-spin terms in the thermodynamic potential, 
which can provide easy or hard axis anisotropy for each 
spin, if one deals with spins larger than 1/2. However, 
such anisotropy can be shown to be determined by high 
energies, e ~ W, and thus cannot be reliably calculated 
in the present approach, designed to capture low energy 
physics of surface electrons. 

We consider a model, in which an individual impurity 
spin interacts with the surface electrons via anisotropic 
exchange Hamiltonian [11 , 



H,. 



J z S z a z S(r- r ) + Jn (S x a x + S y a y ) S(r- r ) 



(4) 

where ro is the position of the impurity, z is the direction 
perpendicular to the surface, and Si is the impurity spin 
operator. Kinetic energy of electrons is described by the 
Hamiltonian Q, with bandwidth W (for Bi 2 Se3 given by 
0.3 eV pQ), and a short range cut-off a = v/W. 

The T-matrix, t(s) is found using the Lippmann- 
Schwinger equation: 



i(e) = V i a + V i aG (s,0)t(e), 
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FIG. 2: Density of states of the topological surface states with 
a band gap induced by ferromagnetic ordering of adatoms. 
Density of states for different concentrations of magnetic 
adatoms. Complete polarization was assumed, and J Z S = 
0.5Wa 2 , W = 0,3eV. 



The expression for the unperturbed Matsubara Green's 
function of the surface electrons, Go(e,r), for the Hamil- 
tonian reads 



(3) G (e,r) = - 



ie 
2?n7 2 



e r 





'|£|r\ 


) - ( 









(r x o%,(7) 



where Kq,i(x) are modified Bessel functions, and f is the 
unit vector in the direction of r. For r — > the above 
equation takes the following form, 



G o (e,f^0)=g(e)a°, g(e) = 
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Using the above relations, we get the form of the T- 
matrix, 
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The above form of the T-matrix exhibits poles at 
energies found from the equation 1 — g 2 V 2 = 0. In 
the limit of a very large bare potential, \V\ >• Wa 2 , 
the resonances are positioned at low energies, |e| ~ 

2nv 2 /\V\ In [T^j . similar to the case of graphene 

(see, e.g., Ref. [Hj). It is easy to see that the impurity 
spin dependent part of the T-matrix vanishes in both 
limits of \V\ -> and \V\ -> 00. Thus the RKKY in- 
teraction reaches a maximum at r ~ |V|/TVa 3> a for 
strong exchange, max( J\\,J Z ) ^> 2ttv 2 /W. This effect is 
missing in the perturbative treatment of RKKY. 

Lowest order perturbation theory [20] can be used for 
moderate exchange values, max(Jii, J z ) < 2nv 2 /W. For 
simplicity, we focus on this case below; RKKY interac- 
tions for the case of strong exchange will be discussed 
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elsewhere [TS]. At moderate exchange, the RKKY inter- 
actions at not too small adatom separation r»a can be 
obtained from the general expression ^ by expanding 
the logarithm to the lowest order. This gives, up to a 
small corrections of the order a/r, 

U 12 (r) = -J 2 ~SfS%-J? l ~(8 1 -^(S 2 -^ + J 2 ~Sis£, 

(9) 

where S^ 2) = S m ■ (f x n), C = ^ J d& 2 (# 2 (£) + 

= 64^' d — T^-jdtem^-Km) = 

J — = (7/2. Therefore, the interactions between two 
impurities, mediated by the surface states, have an un- 
usual, strongly anisotropic form which stems from the 
spin-momentum entanglement on the surface. 

We analyze the collective behavior of adatoms under 
the realistic assumption that their spatial distribution is 
completely random, and the exchange coupling to the 
surface electrons is not too strong, as discussed above. 
The positional randomness combined with the form ([9| 
of the RKKY interactions makes the in-plane interac- 
tions frustrated: the exchange is ferromagnetic between 
components of spins in the plane perpendicular to r, and 
antifcrromagnetic for the components of the spins paral- 
lel to r. Instead, the ferromagnetic interactions between 
z-components of spins can be optimized simultaneously. 
We conclude that for 5 = J\\/J z < 1 the ground state 
of any system of adatoms is a ferromagnet with magne- 
tization along z-axis. In the opposite limit, 5 1, the 
frustrated xy interactions dominate, giving rise to the 
ground state in which spins are frozen in the xy plane, 
such that the average polarization in the z direction van- 
ishes. We expect this phase to be a spin glass, which is 
separated from an Ising-type out-of-plane ferromagnet by 
a quantum critical point. We also expect the ferromag- 
netic ground state to survive some degree of exchange 
anisotropy, such that the critical point corresponds to 
S c > 1. 

As the ferromagnetic ordering breaks discrete Z 2 sym- 
metry, it is separated from a paramagnetic phase by a 
finite-temperature second order phase transition of the 
conventional Ising type. The spin glass phase should not 
exist at finite temperature due to the fact that the surface 
is two dimensional. The phase diagram is summarized in 
Fig.[T] 

The value of 5 C depends on the magnitude of the im- 
purity spin. We choose the most unfavorable for ferro- 
magnetic ordering case of S = 1/2, having the largest 
quantum fluctuations favoring disordered phases. To es- 
timate the value of 5 C , we have performed numerical sim- 
ulations on small spin systems. We exactly diagonalized 
Hamiltonians of randomly distributed spin-1/2 clusters 
with pairwise interactions given by Eq. (J9j) . The resulting 
magnetization, averaged over 150 disorder realizations, is 
illustrated in the inset in Fig. [T] At 5 w 1.3 magnetiza- 
tion decreases by 50%; we take this point to be the finite- 



size approximation to the point of quantum phase tran- 
sition [17] . The position of the transition point depends 
weakly on the number of spins in the cluster. For higher 
impurity spins, we expect S c > 1.3; we leave the deter- 
mination of the precise value S C (S) for future work [15] . 

What is the nature of the disordered phase realized at 
zero T and 5 > 8 C 1 We expect that at 5 — > oo (the only 
interactions are in-plane) the frustrated random interac- 
tion should lead to a spin glass phase [T3] , similarly to the 
case of 2D bimodal XY model [T7] , and 3D dipolar dilute 
magnets [18] . We conjecture that weak ferromagnetic in- 
teraction of z spin components does not destabilize the 
spin glass phase, and it extends to the value of 8 = S c , 
where a quantum phase transition takes place. This hy- 
pothesis is supported by qualitative similarity of our sys- 
tem to the Sherrington-Kirkpatrick model |13j . which de- 
scribes the competition between the non-frustrated ferro- 
magnetic exchange and the frustrated sign-changing in- 
teractions. In the model of Ref. [13] ferromagnetic phase 
is destroyed once the frustrated part of the interactions 
becomes strong enough. Our model almost certainly ex- 
hibits a similar behavior. We are conducting numerical 
work to confirm this hypothesis |15j . 

Now we estimate the ferromagnetic ordering temper- 
ature. At 5 <C 1, the ordering transition is essentially 
that of Ising spins randomly distributed in the plane and 
interacting via 1/r 3 interactions. The ordering temper- 
ature can be estimated as the typical exchange interac- 
tion between two neighboring spins. The result for the 
transition temperature can be read off Eq. ^ by setting 
Si'2 = and r = y/n m , where n m is the concentration 
of impurities. This gives an estimate 

T c (n m )=a J ^n 3 r l 2 , (10) 
v 

where a is a numerical coefficient. Our Metropolis Monte 
Carlo simulations, which take into account long-range 
nature of interactions, confirm this estimate with a 1=3 
0.01. We expect this estimate to hold for 6 not too close 
to 5 Cl and, in particular, for weak exchange anisotropy, 
(5-1. For n m a 2 ~ 1, J z ~ Wa 2 , and W = 0.3 eV (band 
gap of Bi2Se3), we obtain T c ~ 30 K, which is in the 
experimentally observable range. 

We now discuss the effects of the magnetic state of 
adatoms on the spectral and transport properties of the 
Dirac fermions. The ferromagnetic phase is characterized 
by the spontaneous breaking of the time-reversal symme- 
try, and therefore leads to a gap in the spectrum of the 
surface states. The most dramatic signature of the fer- 
romagnetic ordering, detectable in transport, is that the 
behavior near the neutrality point changes from metal- 
lic to strongly insulating. In contrast, both paramagnetic 
and spin-glass phases correspond to a gapless state of the 
Dirac fermions; the randomness of the potential created 
by adatoms manifests itself in the small smearing of the 
average DOS near the Dirac point [TS], similar to the 
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case of graphene [TB] ■ The transport in this case remains 
metallic. 

At the mean-field level, the mass induced by ordering 
of adatoms is given by 



iJ z S. 



(11) 



To explore the effect of positional randomness on the gap, 
we have calculated density of states (DOS) in the self- 
consistent T-matrix approximation |25j . finding that for 
the physically relevant case of not too strong exchange, 
J z < 2ttv 2 /W ', the mean-field result is accurate. The 
DOS obtained using SCTMA is displayed in Fig. [2j Tak- 
ing J z /a 2 sa 300 meV, no 2 sa 0.1, and S = 1/2, we obtain 
the estimate of mass m w 15 meV, which puts it in the 
experimentally observable range. 

An important consequence of a mass in the spectrum 
Dirac fermions is quantized half-integer Hall conductiv- 
ity [21], even in the absence of magnetic field. Such 
anomalous quantum Hall effect (QHE) is a direct con- 
sequence of the parity anomaly and has been predicted a 
while ago [52]) but could not be observed experimentally 
to date. The magnetic ordering on the surface provides 
a way to experimentally observe anomalous QHE. This 
can be done, e.g., in a thin slab geometry, where spins 
polarize in the same direction on the opposite sides of the 
slab. In such a geometry half-integer Hall conductivities 
of the two surface add, giving a quantized Hall conduc- 
tivity e 2 /h, which is observable in the standard Hall-bar 
geometry. An alternative way to realize anomalous QHE 
in topological insulators, proposed recently on the basis 
of the ab initio calculations, involves changing the bulk 
band structure with magnetic atoms [23 . 

Now we analyze the effect of potential impurities on the 
interactions and ordering of magnetic impurities - the 
issue relevant for current experiments, where potential 
disorder is inevitably introduced by chemical doping. It 
is well known in the usual case of disordered metals that 
the impurity averaged exponentially decaying RKKY in- 
teraction does not describe interaction between magnetic 
impurities [24] . Instead, the sample-specific interaction 
in a disordered sample has a powerlaw behavior. 

In our case, disorder also causes exponential decay of 
the averaged RKKY interactions at large distances. The 
relevant length scale is given by (twice) the Green's func- 
tion decay length at e — > 0. In the case of strong non- 
magnetic impurities, this length scale is given by [TJJ [T5] 
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where n p is the density of nonmagnetic impurities, and 
we neglected factors of order of unity. We find [15] that 
RKKY interactions decays exponentially at distances 
r 3* P-dis- The sample-specific RKKY interactions are 
random, have the Dzyaloshinskii-Moriya form, and decay 
as a powerlaw [H] . We thus are led to the conclusion that 



for large densities of nonmagnetic impurities, n p n m , 
the RKKY interaction is primarily of the Dzyaloshinskii- 
Moriya form, and the impurity spins remain magnetically 
disordered down to very low temperatures and/or small 
values of J\\/J z . Thus, magnetic ordering can only be 
observed in relatively clean samples. 

Finally, we discuss the effect of finite doping on the 
magnetic ordering of adatoms. In the doped system, 
the phase diagram discussed above remains essentially 
unchanged for large enough adatom concentration, such 
that adatom separation is smaller than the Fermi wave 

— 1/2 

length, n m <C Xf- At small adatom concentration 
Tim ^ Xf, the Fermi surface effects make RKKY 
coupling sign-changing. In addition, as pointed out in 
Ref. [50]) in this regime the RKKY interaction acquires 
a Dzyaloshinskii-Moriya component, which frustrates in- 
teractions even more. Thus a finite doping stabilizes the 
spin glass phase and suppresses the ferromagnetic order- 
ing. 

In conclusion, we have studied collective properties 
of magnetic impurities on a topological surface. The 
ferromagnetic ordering, occurring in particular for the 
isotropic exchange, provides a way to gap out the sur- 
face state and realize an anomalous Quantum Hall state. 
This requires clean samples, with concentration of po- 
tential impurities being smaller than the concentration 
of magnetic adatoms. 

Recently, we have become aware of an experimental 
work [26] . where deposition of magnetic impurities was 
used as a tool to modify properties of Bi2Se3. Above 
certain density, ferromagnetic ordering, leading to a gap 
opening, was observed, in agreement with our analysis 
above. The suppression of ordering at low densities n m 
observed experimentally is likely due to potential disor- 
der, which stabilizes the spin glass phase, as discussed 
above. 
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